ROUSE and the multinomial model: A priori versus a posteriori predictions

The multinomial model of Ratcliff and McKoon (in press) is an alternative to the Responding Optimally with Uncertain Sources of Evidence (ROUSE) model developed by Huber, Shiffrin, Lyle, and Ruys (in press). The multinomial model, using the ROUSE principles of source confusion and discounting, fits the data collected by Huber et al. (in press) about as well as ROUSE, but is descriptive rather than predictive and must adjust its parameters in accord with each new experiment. We illustrate by showing how ROUSE predicted the results in Huber et al. (in press) as well as three new studies in a priori fashion, whereas the multinomial model requires continuous revision as the new results appear. We prefer ROUSE because it continues to predict new results on the basis of its causal structure. ROUSE and the Multinomial Model 3 ROUSE and the multinomial model: A priori versus a posteriori predictions Huber, Shiffrin, Lyle, and Ruys (in press) presented a theory of short-term word priming termed Responding Optimally with Unknown Sources of Evidence (ROUSE). The theory was motivated by the results of a short-term priming experiment (Experiment 1 in Huber et al.) measuring perceptual identification with two alternative forced choice (2-AFC) testing. In this paradigm, a trial consists of two primes, followed by a brief target flash, followed by a mask, and finally two choice words. In the four basic conditions, the relationship between the choice words and the primes vary: Either the target, foil, neither, or both choice words may be related to a prime word (henceforth labeled target-primed, foil-primed, neither-primed and both-primed). When the prime words were viewed passively (e.g., participants were instructed that prime words were merely a warning to prepare for the brief flash of the target word), performance rose in the target-primed condition and fell in the foil-primed condition, both compared to the neither-primed baseline condition. This pattern constitutes a preference to choose primed words (see Ratcliff, McKoon, & Verwoerd, 1989, Masson, & MacLeod, 1996, or Ratcliff & McKoon, 1997, for early uses of this paradigm in long-term repetition priming). The most striking finding in Huber et al.’s (in press) experiments was a switch in the direction of preference depending upon how the primes were processed: When participants processed the prime words actively (e.g., deciding whether the two prime words could serve the same part of speech) there was instead a preference against choosing primed words (i.e., compared to the neither-primed condition, performance was worse in the target-primed condition and better in the foil-primed condition). This pattern arose clearly when the primes were related to choice words by identity (termed repetition priming). With orthographic-phonemic and associative priming, the ROUSE and the Multinomial Model 4 preference switch between passive and active priming was not as pronounced, but was qualitatively similar. Despite the observed changes in the direction and magnitude of preference, both-primed performance was usually lower than neither-primed performance for both passive and active priming. The ROUSE Model The ROUSE model accounted for these and other findings using conceptually simple and elegant assumptions (in our possibly biased view). The first explanatory principle in ROUSE is termed source confusion (i.e., unknown sources of evidence). Features thought by the participant to have derived from the target presentation can actually be activated by any of three sources: the target presentation, visual noise, or the primes. The second explanatory principle derives from a Bayesian decision process that attempts to correct for this source confusion in an optimal manner (i.e., responding optimally). Estimates of the three possible sources of activation are used to calculate the evidence for each feature of both choice words. The total evidences in favor of each choice word are then compared and the choice word with more evidence is selected. In this manner, features shared with primed words provide a lower degree of evidence (i.e., they are mistrusted), because the prime, rather than the target, may have been the true source of activation. We term this lowered level of evidence discounting. Contrary to claims by Ratcliff and McKoon (in press), we believe this model is at least as simple and certainly more elegant than their multinomial model. There is a sense in which the ROUSE model is indeed more complex than the multinomial model: In certain cases the predictions are not intuitively related to the parameter values, and, in general, the predictions are more complex to derive. We note that in science there are many ROUSE and the Multinomial Model 5 instances in which simple assumptions result in solutions that are difficult to derive (e.g., determining movement of three or more bodies under gravitational forces). Furthermore, we provide, in our articles, both simulation and analytical programs that allow easy generation of predictions, such that, ease of prediction is probably the least important basis for preferring one model to the other. There is a much more important scientific reason for preferring ROUSE to the multinomial model, a reason involving the abilities of the models to produce a priori predictions. To make this case, we start by describing in more detail the causal structure underlying ROUSE and the way this structure leads ROUSE to predict “both-primed deficits”, “preferences”, and “changes in the direction of preference” for passive versus active prime processing. First, consider source confusion. Because it is unknown whether an activated feature derived from the target presentation, noise, or the primes, the extra activation of primed features will by default (i.e., without discounting) result in a preference for primed words. A crucial component of feature activation is its probabilistic nature. In the both-primed condition, on average across trials, the same number of additional features in both the target and foil are activated by the primes. However, on any given trial, more features may be activated by the primes in one choice word compared to the other. This extra variability associated with probabilistically activated features results in both-primed deficits as well as a deficit in the average of the target-primed and foilprimed conditions. Next, consider discounting. In accord with optimal Bayesian decision-making, evidence from activated features is discounted when those features may have been activated by primes (i.e. when the activate feature is shared with a prime). Provided the estimate of prime activation is accurate, discounting may reduce the evidence so as to equate the target-primed and foil-primed conditions ROUSE and the Multinomial Model 6 (i.e., neither a preference for nor against primed words). It is assumed that too little discounting occurs when primes are viewed passively, and too much discounting occurs when primes are processed actively. In many conditions of Huber et al. (in press), this change in discounting reversed the predicted direction of preference. Too little or too much discounting is implemented in ROUSE by assuming the participant's estimate of prime activation is slightly too low or too high. (Participants might also misestimate the two other parameters, governing activation of features by the target presentation and noise, but simulations of such misestimates have shown that this does not change the basic pattern of predictions). It is noteworthy that the ROUSE model, developed following Experiment 1 of Huber et al. (in press), then predicted correctly, in a priori fashion, with consistent parameter values, the results of the subsequent three Huber et al. experiments even though the patterns of those results deviated from that just described. These additional predictions were not at all intuitively obvious consequences of the model (as mentioned earlier). In fact, it took some time after the data were collected before the authors realized that ROUSE made the correct predictions, and considerable additional analyses were needed to determine the reasons for the model's success. We will make an effort in this report to explain heuristically the bases for ROUSE's a priori predictions of these effects. In addition we briefly summarize three new experiments for which predictions were generated in advance of data collection (by now, we know to run simulations in advance, rather than trust our intuitions). In each case, the predictions held true. First, however, we mention a few critical elements of the multinomial model. ROUSE and the Multinomial Model 7 The Multinomial Model Ratcliff and McKoon (in press) presented a multinomial model and fit it to the results reported by Huber et al. (in press), as well as their replication of one condition of another experiment we carried out addressing the role of target duration (Huber, Shiffrin, Quach, and Lyle, submitted). The multinomial model uses a logogen, or word-based representation, rather than ROUSE's feature representation, but has a number of important conceptual similarities to ROUSE: In particular, it borrows the ROUSE principles of source confusion and discounting. Given that it was tailored a posteriori to predict the data set to which it was applied, and used the key principles that Huber et al. (in press) used to produce good fits, it is perhaps unsurprising that the multinomial model also predicted the data. From a larger standpoint, the success of the multinomial approach provides further evidence for the important role that source confusion and discounting play in word identification. Source confusion in the multinomial model occurs within short-term memory. It is assumed that with some probability, prime words take the position of the flashed target word, in which case the choice is mistakenly made based upon the prime word. As with ROUSE, this probabilistic source confusion produces a preference for primed words as well as both-primed deficits. As with ROUSE, discounting is an attempt to correct for source confusion. However, in the multinomial model, discounting results in the total rejection of a primed word, rather than a lowered level of feature evidence. In the multinomial model there is a probability of target perception, a probability of source confusion, and a probability of discounting, and these probabilities are ordered in a multinomial decision tree (discounting taking precedence over source confusion which in turn takes precedence over perception). ROUSE and the Multinomial Model 8 A Priori Versus A Posteriori Predictions We now discuss the manner in which ROUSE and the multinomial model handle the main findings in Huber at al. (in press) and several new studies (Huber, Shiffrin, Quach, and Lyle, submitted; Huber, Shiffrin, Lyle and Quach, in preparation). This discussion illustrates the differential abilities of the models to make testable predictions in advance of data collection. The Efficacy of Discounting The ROUSE model was designed with slightly too much discounting for active priming, and slightly too little discounting for passive priming, to predict the finding from Experiment 1 of Huber at al. (in press) in which the direction of preference reversed between these conditions. At the time, we believed this preference reversal would hold generally given slight underand overestimates of prime activation. However, as we now describe, the predictions for the direction of preference for active priming are a subtle matter, depending on the details of each experiment. It turns out that in several well-specified circumstances, discounting loses its effectiveness, resulting in a preference for primed words even in active priming. ROUSE uses discounting to reduce the evidence assigned to features activated by primes (this could be thought of as “accurate” discounting); however, with unknown sources of evidence, there is no way to prevent the “erroneous” discounting of features shared with primes that were actually activated by the target flash. Discounting these latter features amounts to “throwing the baby out with the bathwater” and it is only through this erroneous discounting that ROUSE can produce a preference against primed words (even very high levels of accurate discounting for prime activated features only serves to reduce, but not reverse preference because, with a ROUSE and the Multinomial Model 9 Bayesian calculation, there is a natural lower bound for a discounted level of evidence). In the case of active prime processing, the level of discounting is slightly too high, such that the erroneous discounting of prime features activated by the target flash produces a preference against choosing primed words. Consideration of the crucial role played by erroneous discounting lead to the following general prediction for the efficacy of discounting: A manipulation serving to reduce sufficiently the probability of primed, but target-activated, features will start mixing in larger number of cases in which no erroneous discounting occurs, thereby increasing preference for primed words. A large enough reduction in erroneous discounting in fact shifts the direction of preference, resulting in a preference for primed words regardless of the extent of discounting. Three variables have been identified and shown to produce exactly this effect (these ROUSE predictions were generated with the standard parameters that produced qualitatively correct predictions for all the studies in Huber et al., in press). These variables are: increased similarity between choice words, decreased target duration, and decreased similarity between primes and choice words. In each case, ROUSE makes an a priori prediction of a preference switch within active priming (but not passive priming) as these variables are manipulated (this amounts to a crossover in the target-primed and foil-primed conditions). We take these predictions up in turn: In each case we contrast the constraint due to the efficacy of discounting in ROUSE with the multinomial model in which discounting can produce a preference in either direction regardless of these manipulations. Similarity of the choice words to each other. When choice word similarity is increased, there are fewer diagnostic features (because features common to the two choice words play no differential role; consider DIED versus LIED). Therefore the probability of erroneous discounting occurring within the small number of relevant features is low. Experiment 3 of Huber et al. (in ROUSE and the Multinomial Model 10 press) demonstrated the predicted crossover in active priming as a strong preference for repeated words emerged with similar choice words. This result posed a problem for simple forms of the multinomial model, so, as explained by Ratcliff and McKoon (in press), the multinomial was amended by assuming that with high choice word similarity, sometimes the wrong choice word is discounted. Target duration. When target duration is reduced toward zero, activation of features by the target flash will grow less and therefore the probability of erroneous discounting of target activated features will drop toward zero. We tested this prediction in a repetition priming experiment (Huber, Shiffrin, Lyle, and Quach, in preparation) by presenting targets for their usual brief duration (full), for half that amount of time, or for zero duration. (This experiment, run prior to the target duration experiment reported by Ratcliff & McKoon, in press, included an active priming condition not used by Ratcliff and McKoon). The verification of the ROUSE predictions is illustrated in Figure 1: In passive priming there was a preference for repeated words that grew larger as target duration dropped, whereas, in active priming, there was the usual preference against repeated words at normal flash duration, but this changed to a preference for repeated words as target duration dropped to zero. This result is handled by ROUSE by reducing the target activation parameter, as predicted a priori, with no changes in the source confusion or discounting parameters. No additional assumptions were made. To account for the passive priming results, Ratcliff and McKoon (in press) modified the basic multinomial model by assuming that source confusion is negatively related to target duration (i.e., source confusion increases as target duration decreases), an assumption that might also let their model handle the active results. Nonetheless, this assumption would almost certainly not have been made a priori. ROUSE and the Multinomial Model 11 ============ Figure 1 here ============ Similarity of primes to choice words. As prime similarity is reduced, fewer features are shared between a prime and a primed target and it becomes unlikely that any erroneous discounting will occur (at the lower limit of zero similarity, there is no discounting at all, such as in the neitherprimed condition). Of course, reducing prime similarity also reduces prime activation, therefore reducing accurate discounting, but because prime activation is consistently estimated to be greater than target presentation activation, erroneous discounting drops out more readily. The a priori ROUSE prediction that a preference against primed words should switch to a preference for primed words, as prime similarity is reduced, was verified in Experiments 1, 2, and 4 of Huber et al. (in press). All of these preference changes were predicted correctly with no change in the source confusion or discounting parameters. We made only the natural a priori assumption that associative similarity was less than orthographic similarity, which in turn was less than repetition similarity. In contrast, the multinomial handled these data by allowing both source confusion and discounting to vary freely with the different types of priming. In order to test this ROUSE prediction even more rigorously, we ran an experiment (Huber, Shiffrin, Lyle, and Quach, in preparation) in which there was face validity to the ordering of prime similarity, and in which there was a single pool of potential target and foil words such that the same parameters should apply to all conditions: For each potential target or foil word, prime words were chosen which shared 4, 3, or 2, of the 5 letters in the same letter-positions (e.g., the choice word MOUND primed by ROUND, MOUSE, or FLUNG). Figure 2 summarizes the data, verifying the ROUSE a priori predictions: The complex interactions of preference with the ROUSE and the Multinomial Model 12 passive versus active manipulation were predicted by ROUSE (the black dots) with common parameter values across similarity conditions with the exception of a single decreasing similarity parameter (as in the Huber et al., in press, experiments, the same similarity values were used across both passive and active priming). We expect that the multinomial model could account for these results, but it would require an a posteriori choice of exactly the right relationship for both source confusion and discounting as a function of prime similarity (i.e. both source confusion and discounting decreasing with decreasing prime similarity, but the effect of discounting decreasing faster than the effect of source confusion). ================== Figure 2 about here ================== The Cause of the Both-Primed Deficit ROUSE and the multinomial model, in its present form, posit testably different causes for the both-primed deficits. For repetition priming, in ROUSE, the both-primed deficit is solely due to source confusion (i.e., probabilistic activation of prime-related features) and the magnitude of the deficit is independent of the level of discounting (i.e., independent of the estimate of prime activation), whereas in the multinomial model, source confusion and discounting are additive factors, both contributing to the both-primed deficit. Therefore, we designed a repetition priming experiment (Huber, Shiffrin, Quach, and Lyle, submitted) to ascertain whether the both-primed deficit would increase when discounting increased, while holding source confusion constant. To do this, we partitioned the results on the basis of the participant's ability to recognize whether choice words also appeared as prime words (i.e., prime recognition). If a choice word was ROUSE and the Multinomial Model 13 recognized as having appeared as a prime word, it was assumed that more discounting took place than if the choice word were not recognized. To insure that a reasonable number of recognition failures would occur, the primes were presented for a relatively short duration. A trial sequence consisted of two primes flashed for 156 ms (on average, across participants), followed by the target flash for 61 ms (on average, across participants). Both primes and target presentations were pattern masked. Following a decision of which choice word was the target, participants rated each choice word for their confidence that the choice word had been presented as a prime on that trial. In analyzing the results, target identification performance was partitioned on the basis of these prime recognition ratings as they applied to primed (repeated) choice words. Because primes were presented for durations well above perceptual threshold, and participants knew they would have to recognize these words, it was assumed that perception and source confusion were constant regardless of prime recognition. It was also assumed that the degree of prime recognition (due to post-perceptual factors like memory failure) determined the degree of discounting. Although it could be argued that these assumptions are not a priori predictions of ROUSE, they were chosen by us in advance of the experiment as the most reasonable possibility, and we ran the experiment because ROUSE and the multinomial model could be discriminated were the assumptions true. As described next, the results did tend to affirm the validity of these assumptions. Figure 3 summarizes the results of this experiment. Neither-primed performance is shown by the bar on the left. Average both-primed performance is the next bar, and shows the usual deficit. This condition is broken down in the next four bars, labeled a through d, based upon high versus low recognition of primed choice words. As predicted, high prime recognition for both choice words and low prime recognition for both choice words produced equal performance (c vs. d). ROUSE and the Multinomial Model 14 When prime recognition for the target and foil differed, discounting differed and produced the expected difference in performance (a vs. b); note that high prime recognition for the target produced high discounting, lowering the target choice probability. ================== Figure 3 about here ================== For the target-primed and foil-primed conditions shown in the right hand bars, recognition of primed choice words likewise resulted in increased discounting; partitioning based on prime recognition (labeled e through h), the difference between the target-primed and foil-primed conditions was large when prime recognition was low (e vs. f), but small and non-significant when prime recognition was high (g vs. h). In other words, with poor recognition of which choice words were also primes, the data look like passive priming, whereas with good recognition of which choice words were also primes, the data look more like active priming. All of these predictions were made a priori. The model’s quite accurate predictions, shown as the dots, were based on only four free parameters: target activation, prime activation, and two estimates of prime activation (one estimate for high prime recognition, and one for low prime recognition). An attempt to fit the multinomial model with the assumption that prime recognition affects only the rate of discounting would necessarily miss the equivalency found between conditions c and d. The multinomial model would incorrectly predict that condition c, with greater discounting, should be lower than condition d. Nevertheless, we expect that the multinomial could be modified a posteriori to fit these data by allowing both the source confusion and discounting parameters to vary freely as a function of prime recognition (changes in one variable exactly ROUSE and the Multinomial Model 15 offsetting changes in the other variable between conditions c and d). Assuming such an exercise were to succeed, it is doubtful that the required pattern of parameter values could have been predicted a priori. Discussion and Conclusions Although we prefer the ROUSE model to the multinomial in good part because it has made and hopefully will continue to make testable a priori predictions that experimentation then verifies, there are other somewhat less important bases for preferring one model to the other. Some researchers might prefer a feature-based approach (ROUSE) to a word-based approach (the multinomial model) because it allows explicit variation of similarity between words through the number of shared features (e.g. shared letters in position). This helped ROUSE correctly predict the preference changes observed as a function of similarity between primes and choice words, and the similarity of the choice words to each other. In the multinomial model there is no direct modeling of similarity. Instead, the manner in which source confusion or discounting should vary as function of similarity manipulations must be empirically determined. Some researchers might prefer the Bayesian formulation that underlies ROUSE for the following reason: With a Bayesian formulation, we can speak of “the optimal” level of discounting and this has a mathematical definition (i.e., when the estimate of prime activation is equal to the actual prime activation). As such it is known exactly how much discounting is appropriate and sub-optimal performance can be qualified as too little or too much discounting. In contrast, there is no clearly defined appropriate level of discounting with a multinomial decision tree; discounting is merely a probability parameter that is free to vary in order to accommodate the results. ROUSE and the Multinomial Model 16 A byproduct of the Bayesian formulation found in ROUSE is a lower bound on the strength of discounting; because discounting occurs in a likelihood ratio, even maximal discounting of active features provides more evidence than inactive features. Because of this constraint, ROUSE made the predictions that, in certain circumstances, the efficacy of discounting would breakdown and a preference for primed words would result even with active priming. Such a constraint on discounting does not exist in the multinomial model. We have been impressed by ROUSE because it made correct predictions that we, its developers, were not initially aware of. In several successive instances we ran an experiment, obtained results we believed inconsistent with the model based on our intuitions, only to discover through simulation, using typical parameters, that the model’s predictions matched the obtained data. Later, we realized the importance of running simulations in advance of data collection, and have run several experiments (specifically the target duration and orthographic prime similarity experiments summarized here) that could easily have falsified ROUSE based upon the preexperiment simulated predictions. In both cases the predictions proved correct. It is, however, hard to find a serious, formal, way to defend this basis for model selection. Although this commentary lists our reasons for preferring ROUSE to the multinomial model, we do not wish in any way to give the impression that we think the multinomial model is of little value. To the contrary, the existence of the multinomial model has already been of considerable value, motivating new studies that have generated important new findings and sharpened out understanding of priming. ROUSE and the Multinomial Model 17